Sensors may be used to generate sensor data indicative of objects in an environment. For example, multiple sensors may provide sensor data to a perception system configured to facilitate autonomous operation of a machine. The perception system may identify a group of objects present in the environment based on the sensor data. However, environmental conditions, damage, miscalibration, and other factors can hinder the effectiveness of one or more such sensors. Additionally, inadequate training data for a particular sensor modality may result in failure of the sensor modality to detect an object disposed in the environment. These situations may be particularly problematic when, for example, an autonomous vehicle operating without an operator relies at least in part on data generated by sensors for proper operation. If one or more of the sensors provides sensor data that is different from and/or that does not otherwise match the group of objects identified by the perception system, the autonomous vehicle may take actions based on inaccurate information related to the environment through which it is travelling. This may hinder operation of such vehicles or machines.